1. Field of the Invention
The present invention relates to a technique for continuously capturing images in a single lens reflex digital camera including a flappable mirror.
2. Description of the Background Art
In a digital camera, a function of continuously capturing images of a subject (so-called a “continuous shooting function”) is performed. That is, images of a subject are continuously captured at a rate of five frames per second, for example.
Image data captured by a digital camera are subjected to various processes for image processing. The various processes for image processing include automatic exposure (which will be hereinafter abbreviated as “AE” process) in which exposure is automatically controlled, and automatic white balance control (which will be hereinafter abbreviated as “AWB” process) in which a white color is automatically adjusted to an appropriate color.
To capture a single frame of image, for example, no significant problem occurs in performing an AE process and an AWB process. Raw data of a captured image is temporarily stored in a main memory, and an evaluation value for an AE process (“AE evaluation value”) or an evaluation value for an AWB process (“AWB evaluation value”) are calculated. Then, exposure can be controlled by adjusting a digital gain for the image data or adjusting a gamma curve in gamma transformation. Also, white balance control can be achieved by adjusting a white balance (WB) gain for the image data.
However, to continuously capture images, there is a problem of a considerable load on a main memory or a bus band. Specifically, after all frames of image data which are continuously captured and remain in a state of raw data are temporarily stored in a main memory and an AE evaluation value and an AWB evaluation value are calculated, the image data is read out from the main memory to be subjected to image processing. Thereafter, the image data is again stored in the main memory. Further, the processed image data is read out from the main memory to be compressed into JPEG format or the like. It is difficult to follow the foregoing sequence in the course of operations for continuously capturing images.
FIG. 17 illustrates the foregoing process sequence. That is, FIG. 17 illustrates a process sequence adapted to a technique for temporarily storing captured image data which remains in a state of raw data in a main memory. More specifically, first, image data output from a sensor is stored as raw data in a main memory 120 via a sensor processing unit (SPU) 104 and main bus 125. Secondly, the raw data stored in the main memory 120 is read out by a real-time processing unit (RPU) 105 via the main bus 125, and is subjected to image processing including exposure control, white balance control, pixel interpolation, and the like. Then, the image data on which pixel interpolation has been performed is again stored in the main memory 120. Further, the image data is read out from the main memory 120 and compressed in a JPEG processor 107, to be stored as JPEG data in the main memory 120. Then, the JPEG data stored in the main memory 120 is sent to be stored in a memory card 122 via a card controller 121.
In a case where processes proceed in accordance with the foregoing process sequence illustrated in FIG. 17, data passes through the main bus 125 at extremely high frequencies, and a large buffer area on the main memory 120 is required. Even in a case where processes proceed under control by direct memory access (DMA) for the purpose of lightening a load on a CPU, the process sequence illustrated in FIG. 17 requires that DMA transfer should be provided six times in total for achieving the following processes:    (1) writing of raw data into the main memory from the SPU;    (2) reading of raw data from the main memory in the RPU;    (3) writing of YUV data to the main memory from the RPU;    (4) reading of YUV data from the main memory in the JPEG processor;    (5) writing of JPEG data to the main memory from the JPEG processor; and    (6) reading of JPEG data from the main memory in the card controller.
Consequently, a large buffer area on the main memory 120 is required, and a considerable load is applied to a bus band of the main bus 125.
FIG. 18 illustrates the capacity of a main memory and the bandwidth of a main bus which are required in the case where processes proceed in accordance with the process sequence illustrated in FIG. 17. Now, consider a situation in which an 8-mega (M) pixel sensor is employed and a continuous shooting rate is five frames per second, for example. Assuming that 2 bytes of data are necessary for each pixel, a bandwidth of 8M×2 bytes×5 frames=80 megabytes/s is required for writing of raw data. Also, the same bandwidth as noted above is necessary for each of reading of raw data in the RPU, writing of YUV data from the RPU, and reading of YUV data in the JPEG processor. Thus, a bandwidth of 80×4=320 megabytes/s is required in order to perform the above-described processes (1), (2), (3), and (4). Further, assuming that the compressibility of JPEG compression is ¼, a bandwidth of 20 megabytes/s is required in order to perform each of the above-described processes (5) and (6). As a result, a bandwidth of 360 megabytes/s is required in order to perform the above-described processes (1) through (6), as illustrated in FIG. 18.
Then, as the number of pixels included in a sensor increases to 10 M, 12M . . . , a required bandwidth increases to 450 megabytes/s, 540 megabytes/s . . . , accordingly. Assuming that the main memory is DDR-166 MHz and the bus efficiency of a 32-bit bus is 50%, the transfer rate of the bus is 644 megabytes/s. As such, when the number of pixels exceeds 16M, it is impossible to complete all processes, as is appreciated from FIG. 18. Likewise, as the number of pixels in a sensor increases, the capacity that the main memory is required to have significantly increases, as illustrated in FIG. 18.
A continuous shooting rate is a very important factor to a single lens reflex digital camera. For example, in shooting a sports scene or the like in which a subject is moving at a high speed, if a continuous shooting rate is five frames per second, the possibility of making the “best shot” is 5/3 times as large as the possibility in a case where a continuous shooting rate is three frames per second. However, to follow the above-described process sequence could not improve a continuous shooting rate because bottleneck of many data transfers with the main memory.
In view of this, the inventors of the present invention has suggested pipeline processing in which image processing is carried out in real time on image data output from a sensor without storing the image data as raw data in a main memory, and the image data is stored in the main memory only after the image data is converted to JPEG data, as taught in Japanese Patent Application Laid-Open No. 2004-304387. In the foregoing manner, that is, by processing data output from a sensor in real time without storing the data in a main memory until the data is subjected to JPEG compression with the use of the technique of pipeline processing, it is possible to lighten a load on a bus band and also reduce the required size of buffer area on a main memory.
Thus, in one possible approach, the above-described method suggested by the inventors of the present invention in Japanese Patent Application Laid-Open No. 2004-304387 is applied to continuous image capture. This approach is expected to save a bus band and considerably reduce the required size of a buffer area in a main memory.
Nonetheless, the foregoing approach causes another problem. Specifically, when data is processed in real time, it is impossible to get times for obtaining evaluation values for an AE process and an AWB process. For the second and subsequent frames of image data, an AE process or an AWB process can be performed by using evaluation values obtained from the first frame of image data in the course of continuous image capture. However, no available evaluation value for an AE process or an AWB process of the first frame of image data exists.
In this regard, in a case where a digital camera is not of a single lens reflex type, but of a compact type, an image is captured by a sensor and the captured image is displayed on a liquid crystal monitor before operations for continuously capturing images start, so that an AE evaluation value and an AWB evaluation value can be obtained before operations for continuously shooting start because.
Unlike this, in a digital camera of a single lens reflex type in which an image is stored in a sensor only after a movable mirror is flipped up, image data is captured only after operations for continuous shooting start. As such, an AE evaluation value and an AWB evaluation value for the first frame of image data cannot be obtained in advance. Accordingly, when pipeline processing is carried out as described above, an AE process and an AWB process of the first frame of image data cannot be performed.
One possible solution to overcome the foregoing problem is to calculate an AE evaluation value and an AWB evaluation value based on image data captured after a mirror is flipped up and to perform exposure control and white balance control on the first frame of image data based on the evaluation values thus obtained. However, this solution has a disadvantage that a period from a time when a shutter button is pressed to a time when image data is actually captured (so-called a “shutter time lag”) becomes longer. Such a long shutter time lag is a significant drawback for a single lens reflex digital camera which is manufactured for use of high amateurs and professional photographers.
In the meantime, various types of single lens reflex digital cameras include a type that is provided with an auxiliary sensor for performing an AE process and an AWB process. Since such an auxiliary sensor is designed to capture image data even before a mirror is flipped up, an AE evaluation value and an AWB evaluation value can be obtained in advance. Nonetheless, such an auxiliary sensor has limited functions, after all, and thus, is insufficient to perform precise exposure control and white balance control. Also, there is a need of mounting an additional component onto a digital camera. On the other hand, an alternative solution in which an AE evaluation value and an AWB evaluation value for the first frame of image data are calculated by using the first half of the first frame of image data and exposure control and white balance control of the latter half of the first frame of the image data are performed by using results of the calculation based on the first half may be thought of. The alternative solution, however, still has a disadvantage that brightness or color varies at some midpoint on a screen, and thus, cannot be employed.